As manufacturing capabilities have improved for image sensor devices, it has become possible to place more pixels on a fixed-size image sensor. As a consequence, pixel size has shrunk. From a signal processing perspective, more pixels imply that the scene is sampled at a higher rate providing a higher spatial resolution. Smaller pixels, however, collect less light (photons) which, in turn, leads to smaller per-pixel signal-to-noise ratios (SNRs). This means as light levels decrease, the SNR in a smaller pixel camera decreases at a faster rate than the SNR in a larger pixel camera. Thus, the extra resolution provided by a smaller pixel comes at the expense of increased noise.
There are several approaches to address the reduced signal provided by ever-smaller sensor pixel size that can result in noisy images. One approach employs image fusion. Image fusion involves acquiring multiple images. These images could come from the same sensor or multiple sensors, they could be of the same exposure or of different exposures, and they could come from different sensors with different types of lenses. Once obtained, the images are spatially aligned (registered), calibrated, transformed to a common color space (e.g., RGB, YCbCr, or Lab), and fused. Due to varying imaging conditions between the obtained images, perfect pixel-to-pixel registration is most often not possible. The problem during fusion then, is to determine if a pixel in an input image is sufficiently similar—via a similarity measure—to the corresponding pixel in a reference image. Fusion performance is directly dependent on the ability of the similarity measure to adapt to imaging conditions. If the similarity measure cannot adapt, fusion can result in severe ghosting. Similarity measures are typically pixel-based or patch-based. Pixel-based similarity measures work well when the reference pixel is reasonably close to the noise-free pixel value. As light decreases and noise becomes progressively comparable to signal, pixel-based similarity measures break down. That is, if the reference pixel is noisy, pixel-based similarity measures use the noisy pixel to decide if the corresponding pixel in an input image should be fused or not. These limitations have been addressed by patch-based distance measures. To decide if a pixel is similar, instead of a single pixel comparison, a patch centered on the pixel to be fused is compared. Typical patch sizes range from 3×3 (9 pixels), 5×5 (25 pixels), 7×7 (49 pixels), and so on. Hence patch-based similarity measures are less sensitive to noise than pixel-based similarity measures. This robustness to noise, however, comes at an increased computational cost: for a 3×3 patch, there are 9 comparisons per pixel as compared to 1 for a pixel-based similarity measure. One challenge then, is to devise methodologies that account for noise so that accurate similarity measures may be developed. With accurate similarity measures image fusion can more readily be used to mitigate a sensor's inherent low signal level.